Problems

1. AgentJanitor cleans up based on the agents_and_hosts table, but it's not complete. There are hosts that' never had an agent. So it should look at the hosts table as well.
2. The order of processing was wrong. Stale hosts should be a subset of unreachable hosts. The current implementation processes unreachable hosts first then stale hosts. This can result in stale hosts being processed twice.

Improvements

Now the AgentJanitor does 3 things:

1. processStaleHosts();
2. determineStaleHostCandidates();
3. cleanUpAgentlessHosts();

The first 2 remain the same, except the order has been swapped. This resolves problem 2. The third task solves problem 1.

cleanUpAgentlessHosts

This method queries the DB via the new API HostDAO:getStaleAgentlessHostIds to get a list of host IDs without associated agents. The list is filtered by last_update so only stale hosts will be process.

SELECT DISTINCT
    hosts.host_id
FROM
    hosts
    LEFT JOIN hosts_and_agents ON hosts.host_id = hosts_and_agents.host_id
WHERE
    hosts.last_update < ?
    AND hosts_and_agents.host_id IS NULL
ORDER BY
    hosts.last_update DESC
LIMIT
    ?

Tests and validations

Deployed to dev1 and manually validated the status by reviewing corresponding logs.

CDP-6636